Update Gemma2 attention scale #694
Conversation
|
Huh, you seem to be correct my bad. I'm not sure why my sanity checks didn't show an enormous divergence |
|
HF implementation uses SPDA so I'd need to switch the attention implementation to access the pattern. I compared the output instead (
|
|
This question is very helpful for me and I am also being confused by this question! How do I adjust the Transformerlens' Gemma to ensure that the Transformerlens Gemma gives the same results as the HF? Which file should I add the above code to? |
|
@mntss Thank you very much for this! @ccp123456 this will be put into a release relatively quickly, so you should be able to make use of this right away. The question of 100% accuracy is a little bit more complicated, and making sure Gemma models are 100% accurate is a pretty high priority task at the moment. If you interested in knowing more, DM me on slack. (If you are not on the slack channel, let me know and I can give you access.) |
Description
Current configuration uses incorrect attention scale. According to deepmind implementation the 2b and 9b versions use sqrt(d_head). This is the default scale in TL.
https://github.com/google-deepmind/gemma/blob/a0504162f99a1c238efb37b8197e711c0f3808fd/gemma/transformer.py#L152-L174
Fixes #693
Type of change
Please delete options that are not relevant.
Screenshots
Please attach before and after screenshots of the change if applicable.
Checklist: